Papers with GPT-2 models
Rethinking Data Mixing from the Perspective of Large Language Models (2026.acl-short)
Copied to clipboard
Yuanjian Xu, Tianze Sun, Changwei Xu, XinLong Zhao, Jianing Hao, Ran Chen, Yang Liu, Ruijie Xu, Stephen Chen, Guang Zhang
| Challenge: | Existing methods to mix data with LLMs have relied on domain definitions derived from intuition. |
| Approach: | They propose a reweighting framework that restructures data scheduling as a graph-constrained optimization problem. |
| Outcome: | The proposed framework achieves competitive performance on GPT-2 models. |
Anchored Answers: Unravelling Positional Bias in GPT-2’s Multiple-Choice Questions (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit a positional bias, particularly an even worse “anchored bias” in the GPT-2 family, where they consistently favour the first choice ‘A’ in MCQs. |
| Approach: | They propose to use the “logit lens” method to trace and modify the internal modules within GPT-2 models responsible for this bias. |
| Outcome: | The proposed approach mitigates the positional bias and improves the accuracy of the GPT-2 model across multiple datasets. |
Transformer Language Models Handle Word Frequency in Prediction Head (2023.findings-acl)
Copied to clipboard
| Challenge: | Prediction head is a crucial component of Transformer language models. Despite its direct impact on prediction, its characteristics have been overlooked in previous analyses. |
| Approach: | They examine the inner workings of the prediction head, specifically the bias parameters, and quantify the effect of controlling their frequency biases on text generation. |
| Outcome: | The prediction head is a crucial component of the Transformer language models. |
WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train large pretrained language models require more computational resources and are expensive to train in other languages. |
| Approach: | They propose a method to transfer pretrained language models to new languages using subword-based tokenization and embeddings. |
| Outcome: | The proposed method outperforms existing methods on low-resource languages and makes training large models more accessible and less damaging to the environment. |
Too Big to Fail: Larger Language Models are Disproportionately Resilient to Induction of Dementia-Related Linguistic Anomalies (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that the attention mechanism in transformer-based NLMs may present an analogue to the notions of cognitive and brain reserve. |
| Approach: | They propose a bidirectional ablation method that masks attention heads to display degradation of similar magnitude to masking in smaller models. |
| Outcome: | The proposed method exhibits properties attributed to the concepts of cognitive and brain reserve in human brain studies. |
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that attention heads have a temporal induction property that allows them to learn and reproduce sequences of tokens. |
| Approach: | They analyze attention heads and transformer outputs to examine in-context temporal biases . they find that transformer output has a tendency toward in-constext serial recall . |
| Outcome: | The findings shed light on similarities and differences between LLMs and human memory and learning. |
Video-Grounded Dialogues with Pretrained Generation Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown success in improving downstream NLP tasks . pre-tuned models capture textual dependencies in text data of rich semantics . |
| Approach: | They propose a framework for improving video-grounded dialogue by extending GPT-2 models . they propose to combine visual and textual representation into a structured sequence . |
| Outcome: | The proposed framework improves audio-visual scene-aware dialogues benchmark on AVSD . it is based on a large pre-trained GPT-2 network and can generate natural responses . |
Disentangling Transformer Language Models as Superposed Topic Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic Modelling is an established area of research where the quality of a given topic is measured using coherence metrics. |
| Approach: | They propose a weight-based approach to search and disentangle decoder-only TLM by a Wikipedia corpus. |
| Outcome: | The proposed approach can be applied to GPT-2 models and LLaMA models. |
Mind the Gap: How BabyLMs Learn Filler-Gap Dependencies (2025.emnlp-main)
Copied to clipboard
| Challenge: | a long-standing debate concerns whether the linguistic input children receive is sufficient to explain the grammatical knowledge they develop. |
| Approach: | They evaluate baby language models trained on child-oriented input from the BabyLM Challenge and two base models trained in 10M and 100M tokens. |
| Outcome: | The proposed models acquire filler-gap dependencies but fail to generalize or fully capture island constraints. |
Language acquisition: do children and language models follow similar learning stages? (2023.findings-acl)
Copied to clipboard
| Challenge: | During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master complex syntactic structures. |
| Approach: | They train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. |
| Outcome: | The proposed model exhibits similar learning trajectories to human children aged between 18 months and 6 years. |
Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study has shown that human-like working memory constraints can be integrated into the Transformer architecture . our model incorporates fixed-width windows and temporal decay based attention mechanisms . |
| Approach: | They propose to integrate working memory constraints into the Transformer architecture . they use fixed-width windows and temporal decay-based attention mechanisms . |
| Outcome: | The proposed models show that they can learn better when training data is scarce . the findings suggest that such constraints may serve as a beneficial bias guiding models towards more robust representations . |